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ABSTRACT 

In 1991, a study was conducted to evaluate intake 
assessment at the various adolescent alcohol and drug treatment 
centers in Washington State. The three goals of this study were: (1) 
investigate the level of agreement among drug treatment professionals 
on the placement of adolescents in different treatment programs; (2) 
evaluate the existing placement process; (3) identify the most 
important factors influencing primary treatment placement decisions. 
Seventy-two assessment files were gathered from various types of 
treatment facilities. Six major findings emerged: (1) the number of 
cases for which consensus was attained was higher in the Team Review 
than in the Independent Review, for both realistic and ideal 
placements; (2) in both the Independent and Team review, there was a 
very high level of correspondence between the realistic and ideal 
placements; (3) in the majority (68/1) of cases in the Independent 
Review, the recommended primary treatment duration was the same for 
realistic and ideal placement; (4) there was £ low level of agreement 
between the panelists' recommended placements and the actual 
treatment the clients received; (5) drug history factors were 
identified as paramount in making placement decision; and (6) little 
standardization of the adolescent assessment process exists. (KM) 
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In the fall of 1991 a study was undertaken on the current intake assessment processes 
used at Washington state's adolescent alcohol and drug treatment centers, by the Adolescent 
Project in the Office of Research and Data Analysis. There were three main objectives for this 
study: 

1) To investigate the level of agreement among drug treatment professionals on the 
placement of adolescents in different treatment programs. 

2) To evaluate the existing placement process. 

3) To identify the most important factors influencing primary treatment placements 
decisions. 



METHODOLOGY 



Fifteen alcohol and drug treatment specialists from around the state were selected to 
participate in this Peer Panel Review. 

Seventy-two client assessment files were selected from 1 10 files gathered around the state. 
Forty-eight assessments came from Regular Outpatient facilities, twelve from Intensive Outpatient 
facilities, and the remaining twelve from Inpatient facilities. 

The assessments were first re /iewed independently by the panelists (Independent Review), 
and then in teams of three (Team Review). For each case, the panelists determined primary 
treatment placement choosing one of three modalities: Inpatient, Intensive Outpatient, and 
Regular Outpatient. They did this for both ideal conditions (that is, given all conceivable 
resources) and realistic conditions (just those resources available in the panelist's region.) 
Additionally, they determined appropriate duration of primary treatment and, from a 
predetermined list of 41 factors, they selected the five most critical in making their placement 
decisions. 



MAJOR FINDINGS AND IMPLICATIONS 



h The number of cases for which consensus was attained was higher in the Team Review 
than the Independent Review, for both realistic and ideal placements* In the Independent 
Review, agreement (weighted) was reached, within a team of three, for realistic placements in 
53% of the cases and for ideal placements in 61% of the cases. (Agreement was defined as all 
three team members agreeing on placement.) The level of agreement increased significantly in 
the Team Review: 87% for realistic placements and 89% for ideal placements (Graph 1). This 
increase in consensus may be a result of the informative nature of the discussions in which key 
aspects of the assessments were highlighted. It may also be a result of group dynamics, with 
individuals gravitating toward majority opinions. 



GRAPH 1: 
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2. In both the Independent and Team reviews, there was a very high level of 
correspondence between the realistic and ideal placements (Graph 2). The high level of 
correspondence between the proposed ideal and realistic placements indicates that the panelists 
are satisfied that the treatment modalities they see as needed for adolescent clients exist in their 
communities. 



GRAPH 2: 
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3. In the majority (68%) of cases in the Independent Review the recommended primary 
treatment duration was the same for realistic and ideal placements. However, the mean 
recommended durations (Table 1) show an increase from realistic to ideal for all three treatment 
modalities: about one month more for Inpatient and Intensive Outpatient. This increase was 
statistically significant (p=0.05) for recommended Inpatient and Intensive Outpatient durations, 
and suggests a need to increase available treatment length for these two modalities. 

TABLE 1: MEAN DURATION OF PRIMARY TREATMENT WHEN 
TREATMENT WAS THE SAME FOR REALISTIC AND IDEAL 
PLACEMENTS, INDEPENDENT REVIEW 



MODALITY 


REALISTIC 
PLACEMENT 


IDEAL; : C l : :V t 
■ PLACEMENT,::; 


PERCENT 
INCREASE 


Inpatient (N=140) 


2.0 months 


3.2 months 


65% 


Intensive Outpatient (N=48) 


5.6 months 


6.6 months 


18% | 


Regular Outpatient (N=49) 


5.2 ^.jnths 


5.6 months 


7% | 
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4. There was a low level of agreement between the panelists 9 recommended placements and 
the actual treatment the clients' received* In the Independent and Team Reviews, the 
panelists* realistic and ideal recommendations were compared to the actual treatment the clients 
received. In all four reviews, 33-37% of the cases actually received the treatment for which they 
were recommended. The majority of cases were recommended for treatment that differed from 
what they actually received. 



GRAPH 3: PERCENT OF PANELISTS' PLACEMENT RECOMMENDATIONS 

THAT CORRESPOND WITH ACTUAL TREATMENT RECEIVED 
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5* Drug History factors were identified as the most important ones in making placement 
decisions. The panelists rank ordered the five most important factors in making their placement 
decisions. These factors were grouped into seven categories: Drug History, Family History, 
Medical Concerns, Social Background, Personal Development, High Risk Behaviors, and Other. 
Drug History factors were selected most frequently for the Independent and Team Reviews, for 
both ideal and realistic placements. In both reviews, Personal Development and Medical 
Concerns were identified least often. In 79% of the cases, the panelists cited the same factors 
for realistic and ideal placements. Table 2 presents the distribution of factors selected by the 
panelists for the independent ideal review. 
(15 panelists x 24 cases x 5 factors = 1800 total factors selected). 

TABLE 2: DISTRIBUTION OF FACTORS, BY CATEGORY, FOR 
INDEPENDENT IDEAL REVIEW 



• 


Independent 
..'Realistic Placemen!!! 


Ik * 

Idea! Placement 


Drug History 


37% 


34% 


Family History 


17% 


18% 


High Risk Behavior 


14% 


13% 


Social Background 


11% 


10% 


Personal Development 


3% 


2% 


Medical Concerns 


2% 


3% 


Other 


7% 


8% 


No Response 


10% 


13% 


TOTAL 


100% 


100% 



6. Presently there is little standardization of the adolescent assessment process. In an 
evaluation completed after the meeting, 88% of the panelists stated more seminars and 
training geared toward standardizing assessments and intakes would be helpful. Likewise, an 
agreement level of 53% (Graph 1) further supports the need for greater standardization. 



eric 



v 



CHAPTER I: INTRODUCTION 



Treatment of adolescents for chemical dependencies is a relatively new field. Little 
information is currently available on the methods and trends of assessment practices in this 
field (Winters, 1990). A survey of over 70 adolescent chemical dependency facilities found 
considerable variation in assessment processes, and a heavy reliance on clinical judgement 
(Owen and Nyberg, 1983). 

As the adolesceni treatment field grows, there may be greater diversity among 
professionals which could lead to greater variability in placement decisions. Additionally, 
expansion of the field may increase the variety of treatment options available. This, as well, 
could lead to greater variability in placement decisions. 

This study was undertaken to examine the assessment process in Washington state's 
adolescent alcohol and drug treatment programs. The following questions were addressed. 

1) Do professionals agree on the appropriate choice of treatment for adolescents? 

2) Does the presence of results from standardized assessment tools (such as the 
Personal Experience Inventory) affect this level of agreement? 

3) If a treatment specialist could place a client in an ideal treatment facility, 
would this differ from their realistic placement? 

4) What is the level of professional agreement on the length of primary treatment? 

5) What factors are involved in making a treatment placement decision? 

6) How well is the current system placing adolescents into treatment programs? 

To answer these questions, a panel of 15 Washington state treatment specialists was 
established. The panelists, independently and in teams, reviewed 72 assessment files from 
treatment facilities around the state. For each case, the panelists recommended a treatment 
placement choosing one of three modalities: Inpatient, Intensive Outpatient, and Regular 
Outpatient. They also recommended the duration of treatment, and selected the five most 
important factors influencing their placement decision. 
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A: METHODOLOGY 



Fifteen alcohol and drug treatment specialists from around Washington State were 
selected to participate in the Peer Panel Review. The 15 panelists were divided into five 
teams of three, to review cases and recommend treatment placements. 

The assessments of 72 cases were selected from around the state for the panel review. 
Twelve of these cases were "common cases", i.e. reviewed by all 15 panelists. The remaining 
60 were "team-specific" cases, i.e. each case was reviewed by only one 3-person team. The 
cases were actual DASA-funded (Division of Alcohol and Substance Abuse) clients, though 
the panelists were not told what type of treatment the clients received. 

Each panelist independently reviewed 24 cases* Members of the same team reviewed 
the same set of 24 cases. Following their Independent Reviews, the panelists met with their 
teams for two days to discuss the cases (Team Reviews). 

In both the Independent and Team Reviews, panelists were asked to determine 
appropriate primary treatment placement under both realistic and ideal conditions. Panelists 
were instructed to define realistic conditions as the current circumstances present in their own 
region. Thus, each panelist had a different framework with which they approached their 
realistic placement decisions. Ideal conditions were defined as having all resources readily 
available, no financial constraints, and space available in all programs. The panelists also 
decided appropriate primary treatment duration. 

Prior to the Independent Review and Peer Panel Review, a procedural meeting was 
held with the panelists. At this meeting the panelists generated a list of 74 factors that 
influence treatment decisions. They divided these factors into seven categories: Drug History, 
Family History, High Risk Behaviors, Social Background, Personal Development, Medical 
Concerns, and Other (miscellaneous). Following this meeting, the panelists were sent the list 
of 74 factors, and asked to identify the 35 factors most critical in making placement 
decisions. The results were tallied, and the top 41 were included in the Independent 
Assessment form completed by the panelists for each case. The 41 factors were distributed 
across the seven categories as follows. 
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TABLii) 2 • NUMBER OF FACTORS IN EACH CATEGORY 





FACTORS 


Drug History 


11 


Family History 


6 


High Risk Behaviors 


5 


Social Background 


6 


Personal Development 


5 


Medical Concerns 


5 


Other 


3 



See Appendix A for a complete list of the factors. 



B: SELECTION OF ASSESSMENT CASES 



One hundred and ten assessment cases were gathered from 23 treatment facilities around 
the state. (See Appendix B for a list of the treatment facilities.) All assessments had been 
conducted in 1991, and all cases were at least partially funded by the Division of Alcohol and 
Substance Abuse. Seventy-two cases were selected from the 110 that were collected. An 
informal selection process tcok several factors into account. These factors were: 

1. Regional/Statewide Selection, 

2. Socio-Demographic Characteristics, 

3. Treatment Modalities; and 

4. Drug History and Use Patterns. 




1, Regional/Statewide Selection: 



The cases selected were from the six different regions in Washington, Table 2-2 presents 
the number of cases reviewed from each region, the percent of the total sample reviewed from 
each region, and the percent of actual adolescent DASA clients from each region. (Clients from 
three western Washington regions were over-represented in the sample as compared with actual 
DASA clients.) 



TABLE 2-2: REGIONAL DISTRIBUTION OF CASES 



Region of WA Stated 


Distribution of 
Actual DASA clients * 


Numbepof ; : 


Percent of j| 

/.jcoj&i^ 


I 


12% 


3 


4% 


II 


20% 


6 


8% 


in 


7% 


1 


2% 


IV 


28% 


29 


40% 


V 


17% 


15 


21% 


VI 


16% 


18 


25% 


Total 


100% 


72 


100% 



* Estimates from Client Descriptive Study, Adolescent Project, Office of 
Research and Data Analysis, DSHS, 



2. Sociodemographic Characteristics: 



In the sample cases, there were 47 male clients and 25 female clients. 41% of DAS A 
clients are female/ Table 2-3 presents the age distribution of actual DASA clients, and the 
number and percent of cases reviewed by the panelists, by the clients' ages. 



TABLE 2-3: AGE DISTRIBUTION OF CASES 



Clients* Ages 


Distribution of 


Numy r of Cases 


-7 

Percent 


Actual dasa clients.;;*;;;;;; 






< 13 


9% 


4 


5% 


14-15 


34% 


25 


35% 


16-17 


45% 


38 


53% 


18 


8% 


5 


7% 


Total 


96% 


72 


100% 



* Estimates from Client Descriptive Study, Adolescent Project, Office of Research 
and Data Analysis, DSHS. Total does not add to 100%, because of 4% unknown. 
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3. Treatment Modalities: 



Proportional numbers of cases of adolescent clients were selected from the different 
treatment modalities relative to those which currently exist for DASA adolescent clients in 
treatment around the state. In DASA facilities, statewide, client distribution by modality is of 
the order: 67% Regular Outpatient, 14% Intensive Outpatient, and 19% Inpatient 



TABLE 2-4: TREATMENT MODALITY DISTRIBUTION OF CASES 



• ' III 


Client 

iMtribution 
5^|jtiui DASA 
Facilities 


Cases If 
Reviewed by 
Entire Pane* 


Cases 

Reviewed by 
Each Panelist 




(1990) * 


^% ^ (N) 




Inpatient 


19% 


17% (12) 


17% (4) 


Intensive 
Outpatient 


14% 


17% (12) 


17% (4) 


Regular 
Outpatient 


67% 


67% (48) 


67% (16) 


Total 


100% 


100% (72) 


100% (24) 



* Estimates from Client Descriptive Study, Adolescent Project, Office of Research 
and Data Analysis, DSHS. 

4, Drug History and Use Patterns: 

Different client drug history and use patterns were selected. In choosing the 72 cases, an 
informal and unsystematic attempt was made to include cases ranging in severity from 
experimental use to addiction. Consideration was given to the tvge of drugs used, length of use, 
amount used, and pattern of use. 

Once the 72 cases were selected, all client identifying information was deleted from each 
client's assessment form. This included: name of client, family and friends, address(es), schools 
attended, and name of facility from which the case was selected. In addition, treatment 
recommendations, counselors' impressions, and diagnoses were deleted. This was done to ensure 
that only objective, factual information was provided to the panelists. Thirty-one of the cases 
included summaries from the Personal Experience Inventory (PEI). The 72 cases were distributed 
in such a way that no panelist evaluated his or her own facility's assessment forms. 
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C: SELECTION OF PANELISTS 



The six DASA Regional Administrators provided suggestions for possible panelists from 
each of their regions. They were asked to consider length and breadth of experience, and 
professional reputation, in making their suggestions. The desire was tc include professionals with 
counseling and intake assessment experience. Many of the professionals have administrative 
experience as well. The six Regional Administrators provided a total of 33 names. 

One assumption of the Peer Panel Review is that the fifteen panelists are representative 
of the professionals working in the adolescent treatment field. To best achieve this, several 
factors were considered in selecting the final fifteen candidates from the regional administrators' 
33 suggestions. 

1. Representation of the Different Adolescent Treatment Modalities: Inpatient, 
Intensive Outpatient, and Regular Outpatient: Of the fifteen panelists, five worked 
in Inpatient facilities, four in Intensive Outpatient, and three in Regular Outpatient. One 
worked in a school setting, one in the prevention division of a County Health Department, 
and one in a MICA (Mentally 111 and Chemically Addicted) program. 

2. The Variety of Employment Positions for Treatment Professionals: Five of the 
panelists worked primarily as treatment counselors, eight were treatment program 
administrators (treatment and program directors, and admissions coordinators), one was 
a school counselor, and one worked as an administrator in a County Health Department. 

3. Experience Working in More Than One Treatment Facility and Modality: 87% had 

experience working in more than one treatment facility and modality. 

4. A State-Wide Regional Distribution: The panelists represent all six state regions, and 
come from nine different counties, including four from eastern Washington, 

5. A Balanced Number of Men and Women: There were eight women and s^ven men on 
the panel. 

6. Representation of Different Ethnic and Minority Populations: Two of the panelists 
were African-Americans, and one was Native American. The rest were Caucasian. 

The five teams of three were composed in a way to most equally distribute the panelists 
by gender, region, and the treatment modality in which they work. 
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D: MODERATORS AND OBSERVERS 



A moderator was assigned to each team of three peer panelists. The role of the moderator 
was to facilitate the discussions and record the treatment placement decisions. Moreover, the 
moderators were responsible for ensuring that all panelists had an opportunity to speak, and that 
the time was closely monitored so all cases would be reviewed within the allotted time. The 
moderators were instructed to remain impartial and objective in their approach. Four of the 
moderators are professionals in the field, and the fifth is a Research Analyst with the Adolescent 
Project in the Office of Research and Data Analysis, DSHS. 

In addition to the moderators, each team also had an observer. The observers were 
responsible for objectively recording the nature of the discussions. Because objectivity was 
crucial to their role as an observer, none of the them was working or involved in the field of 
substance abuse. They were graduate students and college staff selected from the University of 
Washington, Seattle and The Evergreen State College, Olympia. They received a half-day 
training session in which observational techniques were taught and. practiced. They also 
participated in a half-day debriefing session after the Peer Review in which they shared their 
observations about the content and nature of the discussions. In addition, they wrote extensive 
notes describing the discussions that ensued. 



SIGNIFICANCE 

All tests in the report were run at p=0.05 using t-tests or X 2 tests. See Appendix E for 
the list of tests run for each comparison. 
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A: MAJOR FINDINGS 



o Consensus (weighted) increased significantly from the Independent Review to the Team 
Review, for both realistic and ideal placements. For realistic placements, consensus 
(weighted) increased from 53% to 87%; for ideal placements, it increased from 61% to 
89 percent 



o In 81% of the Independent Review cases, panelists recommended the same treatment 
under ideal and realistic conditions. For the Team Review, 87% of the cases were 
recommended for the same treatment under both conditions. 



o The recommended duration of treatment increased significantly from realistic to ideal 
placements for Inpatient and Intensive Outpatient, For Regular Outpatient, recommended 
duration remained the same. 



o The majority of clients (77%) recommended by the panelists for Regular Outpatient 
treatment were actually placed in this modality. However, few of the clients (26%) 
recommended for Inpatient treatment actually received Inpatient treatment, and even fewer 
(17%) of the clients recommended for Intensive Outpatient treatment received this 
treatment. 

o Drug History factors were identified as most important in making placement decisions, 
followed by Family History factors and High Risk Behavior factors. 
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B: LEVEL OF AGREEMENT AMONG PANELISTS 



Each panelist reviewed 24 cases, first independently and then in teams of three. In both 
the Independent and the Team Review, the panelists determined treatment placement under both 
ideal and realistic conditions. Recommendations were evaluated for level of agreement within 
each team as follows: 



1, Complete Agreement or Consensus: all three team members recommended the same 
treatment modality for a given case. 

2. Majority Agreement: two of the three team members recommended the same modality, 
and the third differed in his or her recommendation. 



3. No Agreement: all three panelists recommended different treatment modalities. 

Weighted values were used to account for the probability of obtaining complete 
agreement, majority agreement, and no agreement. The weighting procedure was used because 
the probability of reaching complete agreement among three panelists in a team is much less than 
reaching majority or no agreement. (See Appendix C for a description of the weights and a 
report of the raw numbers.) 



Agreement in the Independent Review 



When determining realistic placements, complete agreement was reached in 53% of the 
weighted cases in the Independent Review. For ideal placements, complete agreement was 
reached in 61% of the weighted cases. Panelists reached majority agreement less frequently for 
both realistic and ideal placements, 30% and 24% respectively (Graph 3-1). 

GRAPH 3-1: DEGREE OF CONSENSUS AMONG PANELISTS IN THE 

INDEPENDENT REVIEW (WEIGHTED CASES) 



Realistic Recommendations 
(n=120) 



Ideal Recommendations 
(n=120) 



D CompIeU Agreement 
H Majority Agreement 
D No Agreement 



Agreement in the Team Review 



In the Team Review, the panelists reached consensus in 87% of the weighted realistic 
placements, and 89% of the weighted ideal placements. They reached majority agreement less 
frequently: 11% of the weighted realistic placements, and 8% of the weighted ideal placements 
(Graph 3-2). 



GRAPH 3-2: DEGREE OF CONSENSUS AMONG PANELISTS IN THE TEAM 

REVIEW (WEIGHTED CASES) 



Realistic Recommendations 
(n=120) 



Ideal Recommendations 
(n=120) 




G Complete Agreement 
M Majority Agreement 
CD No Agreement 
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89% 
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Comparison of Agreement in the Independent and Team Reviews 



Complete agreement (weighted) increased significantly from the Independent Review to 
the Team Review. Complete agreement increased for both realistic and ideal placements, from 
53% to 87%, and 61% to 89% respectively (Graph 3-3). After meeting and discussing the cases, 
the panelists, within their teams of three, more frequently recommended the same treatment. 



GRAPH 3-3: 



COMPARISON OF RATES OF COMPLETE AGREEMENT FOR 
PLACEMENTS, AMONG PANELISTS IN INDEPENDENT AND 
TEAM REVIEWS 
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Note: Because these bars show only complete agreement, they do not add to 100 percent. 

In the group neeting held after the Team Review, the panelists provided possible 
explanations for these findings. They suggested that the increase in agreement was due largely 
to the informative nature of the discussions. By presenting and discussing their ideas, the 
panelists felt they were able to gain a more comprehensive and thorough understanding of the 
individual cases. However, when the team recommendations for individual cases are examined, 
another explanation is suggested. In cases that were reviewed by all teams (common cases) intra- 
group agreement was often reached (consensus within the groups of three), while inter-group 
agreement (between the five teams of three) was not often reached. This suggests that group 
dynamics (the tendency of an individual to agree with the majority) may have played a larger 
role in the level of consensus than did a new understanding of the case. 
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In addition to the disparity across teams in their recommendations for a particular case, 
the overall distribution of individuals' recommendations further suggests that group dynamics 
played an important role in the Team Reviews. Although there was a significant increase in the 
level of consensus after the Team Review, the distribution of placements across the three 
modalities changed very little. Table 3-1 presents the percent of cases placed in each treatment 
modality. There was no statistically significant difference between the Independent and Team 
distributions of recommendations. For example, the percentage of ideal recommendations for 
Inpatient did not increase significantly from Independent (48%) to Team (54%). 



TABLE 3-1: PERCENT OF CASES PLACED IN EACH MODALITY 
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Probability of Three Panelists Agreeing 



To further explore the level of agreement, the panelists' recommendations were studied 
by modality. When a panelist recommended a treatment modality, the probability of the other 
two panelists on the team agreeing with that recommendation was analyzed by the recommended 
modality (Graph 3-4). 

Of the three possible recommended modalities, the probability of reaching agreement 
among three panelists for Inpatient placement recommendations was consistently the highest. 
That is, when a panelist recommends Inpatient in the Independent Realistic evaluation, there is 
a 0.43 probability that the other two panelists will agree; whereas, if a panelist recommends 
Intensive Outpatient or Regular Outpatient for realistic placement, there is only a 0.04 and 0.16 
probability respectively that the other two panelists will also recommend those modalities. While 
the probability of reaching complete agreement increases for all three modalities from 
Independent to Team Reviews, when Intensive Outpatient is recommended, the probability of two 
other panelists agreeing with that modality is consistently lower than for Inpatient and Regular 
Outpatient. This difficulty in reaching complete a ^reement suggests that there may be some 
ambiguity in the definition of Intensive Outpatient treatment programs. 



GRAPH 3-4: PROBABILITIES OF REACHING COMPLETE AGREEMENT BY 

RECOMMENDED TREATMENT MODALITY 
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To more closely examine the different levels of agreement, the Independent Ideal 
recommendations were examined by modality. When a modality was recommended, the 
probabilities that the two other panelists agree (complete agreement), only one other panelist 
agrees (majority agreement), or neither of die two panelist agrees (no agreement) is shown by 
modality in Graph 3-5. 

When Inpatient is recommended, there is a 0.54 chance that the two other panelists will 
concur, there is a 0.32 chance that only one of the other two panelists will agree. Combining 
these two probabilities, when Inpatient is recommended, there is an 0.86 (0.54 + 0.32) chance 
that at least one of the other two panelists will also recommend Inpatient treatment. There is 
only a 0.14 probability that the other two panelists will disagree. The probabilities of the two 
other panelists agreeing when Intensive Outpatient or Regular Outpatient is recommended is 
much lower (0,05 and 0,06, respectively) than when Inpatient is recommended. This might 
suggest that there is less agreement regarding the placement of adolescents in outpatient treatment 
programs than in inpatient treatment programs. 



GRAPH 3-5: PROBABILITIES OF REACHING SOME LEVEL OF AGREEMENT 

BY RECOMMENDED IDEAL TREATMENT MODALITY IN THE 
INDEPENDENT REVIEW 
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Agreement for Cases Wi<* and Without the Personal Experience Inventory (PEI) 



The PEI is a standard assessment tool that was required for all intake assessments in the 
last biennium. Though no longer required, many treatment facilities continue to use it. The 72 
cases were selected irrespective of inclusion of the PEL Random selection resulted in 43% of 
the cases including PEI results. For those cases with PEI results, only the summary sheet was 
provided to the panelists. 

Graph 3-6 illustrates the level of agreement (weighted) among team members for cases 
with and without PEI results. As with graphs 3-1 through 3-3, weighted values were used to 
account for probability of obtaining complete agreement. In both the Independent and Team 
Reviews, there was no statistically significant difference in the level of agreement for cases with 
and without the PEI. The presence of the PEI in a case does not affect the level of professional 
agreement (Unweighted analysis is in Appendix C.) 



GRAPH 3-6: 



LEVEL OF AGREEMENT FOR IDEAL PLACEMENT AMONG 
TEAM MEMBERS FOR CASES WITH AND WITHOUT THE PEI 
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During the Team Reviews, the moderators asked the panelists if the PEI information was 
helpful in making placement decisions for those cases with PEI results. In 48% of the cases, the 
panelists said the results were helpful. 
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Agreement Between Realistic and Ideal Placements 

Comparing individual treatment recommendations for realistic and ideal placements 
showed a high level of correspondence, in both the Independent and Team Reviews (Graph 3-7). 
In 81% of the cases in the Independent Review, the panelists recommended the same treatment 
for realistic and ideal placements, whereas in the Team Review, 87% of the cases were 
recommended for the same treatment. This may suggest that most services are available across 
the state; therefore, there is little difference between what should ideally take place, and what 
would be realistically recommended. However, this may also suggest that the panelists were not 
differentiating between realistic and ideal as rigorously as they actually would in the field. 



GRAPH 3-7: PERCENT OF CASES RECOMMENDED BY PANELISTS FOR THE 

SAME TREATMENT FOR REALISTIC AND IDEAL 
PLACEMENTS 
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When the panelists were queried about this in the group meeting, they explained that often 
the difference between the two placements was not in the treatment modality selected, but in the 
duration of treatment suggested (ideal almost always being longer). Thus, they ordinarily can 
place a client in their treatment of choice, but for less time than would be desired (see "Level 
of Agreement on Placement Durations" in this section). In addition, they suggested that clients 
living in Western Washington had nearly all resources currently available to them, whereas those 
in Eastern Washington had fewer options available. 
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Regional Differences in Agreement for Realistic and Ideal Placements 

Following the panelists' suggestion, we looked for regional variations in the rate of 
agreement between realistic and ideal placement recommendations. The panelists were divided 
by the region of the state from which they came. Four panelists work in Eastern Washington, 
and the remaining 1 1 work in Western Washington. If all treatment options were not available 
in Eastern Washington, then it would be expected that the four that work east of the mountains 
would have less correspondence for realistic and ideal placements, than the 11 west of the 
mountains. However, Graph 3-8 shows there was no statistically significant difference between 
the two regions. The four panelists in Eastern Washington selected the same treatment for 
realistic and ideal placements in 87% of the cases, whereas those in Western Washington selected 
the same treatment in 79% of the cases. 



GRAPH 3-8: REGIONAL DIFFERENCES IN THE PERCENTS OF 

CORRESPONDENCE BETWEEN REALISTIC AND IDEAL 
PLACEMENTS IN THE INDEPENDENT REVIEW 
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The panelists were then divided by metropolitan and non-metropolitan work locations. 
Again no difference was found. Both groups recommended the same treatment for realistic and 
ideal conditions in 81% of the cases. We are unable to confirm geographic differences in the 
level of agreement between the realistic and ideal placement recommendations. 




Level of Agreement on Placement Durations 



When the panelists reviewed the assessments in the Independent and Team Reviews, they 
were asked to determine the duration of primary treatment. The overall mean recommended 
duration of treatment for realistic placements was 3.2 months. For ideal placements this 
increased one month to 4.2 months. 

Table 3-2 shows the recommended mean duration of treatment by modality for those cases 
recommended for the same treatment under realistic and ideal conditions in the Independent 
Review. (Note: 81% of the cases in the Independent Review were recommended for the same 
treatment modality- Graph 3-7.) 



TABLE 3-2: MEAN DURATION OF TREATMENT WHEN TREATMENT WAS 
THE SAME FOR REALISTIC AND IDEAL PLACEMENTS, 
INDEPENDENT REVIEW 



TREATMENT ' 


REALISTIC 


IDEAL 


PERCENT 


MODALITY 


"PLACEMENT 


PLACEMENT 


rlNqRE^Iij 


Inpatient (N=140) 


2.0 months 


3.2 months 


65% 


Intensive Outpatient (N=48) 


5.6 months 


6.6 months 


18% 


Regular Outpatient (N=49) 


5.2 months 


5.6 months 


7% 



The mean recommended duration increased for all three treatment modalities from realistic 
to ideal. The increase was statistically significant for Inpatient and Intensive Outpatient treatment 
only. This suggests the need for an increase in the length of treatment available for these two 
modalities. 

Table 3-2 also shows that clients recommended for Intensive Outpatient treatment were 
recommended for the longest duration of treatment, for both realistic and ideal placements. 
Inpatient recommendations were for the shortest length of time. This is similar to actual 
treatment durations, where Inpatient clients generally receive one or two months of primary 
treatment, and Outpatient treatment tends to be for several months. 
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In 68% of the cases in the Independent Review, the recommended primary treatment 
duration was the same for realistic and ideal placements. However, the level of agreement 
between realistic and ideal treatment duration differed by modality. As Graph 3-9 illustrates, 
there was less agreement between realistic and ideal placements on treatment duration for 
Inpatient clients than for Intensive and Regular Outpatient clients. For example, in 49% of the 
cases recommended for Inpatient treatment, the recommended duration was the same for realistic 
and ideal placement. 



GRAPH 3-9: LEVEL OF AGREEMENT ON TREATMENT DURATIONS FOR 

REALISTIC AND IDEAL PLACEMENTS (INDEPENDENT 
REVIEW) 
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In all but one case, when the duration of treatment for Realistic and Ideal placements did 
not agree, the panelists recommended longer treatment for Ideal placements. Graph 3-9 shows 
that differences in the prescribed ideal and realistic duration recommendations for Intensive 
Outpatient are infrequent. The statistically significant increase (of 18%) from realistic to ideal 
placements (Table 3-2) for Intensive Outpatient recommendations is the product of a few cases 
with a large difference in duration, while the majority of duration recommendations for Intensive 
Outpatient is the same for realistic as ideal. 
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Level of Agreement for Common Cases 



Twelve cases were reviewed by all 15 panelists. By looking at these common cases, a 
better measure of agreement in the field can be obtained. Instead of focusing on the level of 
agreement of a team of three panelists, significant agreement within the entire group of fifteen 
panelists is examined. Agreement among fifteen panelists provides a more reliable measure of 
agreement, than among three panelists. 

Significant agreement was determined by comparing the distribution of the fifteen 
panelists' individual case recommendations to the average distribution for all common case 
recommendations in the Independent Review. (See Appendix D for the distribution of 
recommendations for all 12 common cases.) 

Graph 3-10A shows that for realistic placements in the Independent Review, panelists had 
a significant level of agreement in 42% (five out of twelve) of the cases. For the ideal 
placements, there was significant agreement in 50% i,six out of twelve) of the cases. Thus, there 
was slightly more agreement in the ideal placements than in the realistic placements. For the 
Team Review, the percentage of cases in which significant agreement was reached increased to 
67% (8 out of 12) for both realistic and ideal placements. 

These increases are similar to those observed when consensus within the teams of three 
was investigated. Overall, however, the percentages are somewhat lower than those obtained by 
analyzing consensus within the teams of three. This is not surprising, given that it is more 
difficult to reach a significant level of agreement in a group of fifteen than it is to have 
consensus in a team of three. It is also important to recognize that agreement is being measured 
in two different ways: within teams of three and among the fifteen panelists. 



GRAPH 3-10A: 
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Note: "Significant Agreement" among the panelists was determined by using multiple t tests (p=0.05) on 
the distribution of fifteen placement recommendations for each common case. 
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Common cases were also analyzed by the level of pairwise agreements there were among 
the fifteen panelists. That is, for each case, we looked at how many times two panelists agreed 
with each other on any modality. There are 105 possible ways any 2 panelists in a group of 
fifteen can agree. Graph 3-10B shows the percentage of pairwise agreements obtained (regardless 
of modality) out of the 1260 (105 x 12 cases) possible pairwise agreements. These results are 
very similar to those observed in Graph 3-10A. 



GRAPH 3-10B: PERCENTAGE OF PAIRWISE AGREEMENTS IN COMMON 
CASES 
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These two graphs, 3-10A and 3-10B, show similar levels of agreement to that found when 
examining weighted agreement within teams of three. 
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C: APPROPRIATENESS OF ACTUAL PLACEMENTS 



Information on how the 72 clients were actually placed was known, but not shared with 
the panelists. Comparing recommended treatment with actual placement serves as a proxy 
measure for the appropriateness of actual treatment. If a high level of concordance was achieved, 
this would suggest the current system closely approximates the panelists' recommended treatment. 
Thus actual placements could be regarded as appropriate. 

It is important to note, however, that actual recommended treatment is not known, but 
actual received treatment is. Many factors effect the eventual treatment a client receives. 
Recommended treatment is only one of these factors. In addition to this, actual placement will 
be strongly influenced by availability of a particular program, as well as willingness of the client 
to enter a particular treatment. The panelists reported a high refusal rate by adolescents to enter 
Inpatient treatment. Often, teens are placed in Intensive Outpatient programs because of their 
resistance to Inpatient settings. Thus, caution must be exercised when comparing panelists' 
recommendations to actual placements. 

Comparison of Recommended Placements with Actual Treatment Received 

In the Independent and Team Reviews, the placement recommendations made by the 
panelists were compared to the actual treatment the clients received. All recommendations for 
"Other" treatment (not Inpatient, or Intensive or Regular Outpatient) were excluded from this 
analysis. Graph 3-11 shows that for all realistic recommendations made h the Independent 
Review, 35% were for the same treatment the clients actually received. In other words, 65% 
were recommended for treatment that differed from what the clients actually received. When 
panelists recommended the clients for ideal treatment in the Independent Review, again there was 
little agreement with the actual treatment the client received (37%). Similar results were found 
in the Team Review. 

While agreement between recommended placement and actual treatment seems low, this 
may be due in part to the other factors mentioned above which affect actual placements, such as 
client refusal. 
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Correspondence Between Recommended Placements and Actual Treatment, by Modality 



As was shown in Graph 3-11, 35% of the realistic recommendations made in the 
Independent Review corresponded with the actual treatment received by the clients. Differences 
were observed for the three treatment modalities, when panelists' recommendations were 
compared to actual treatment received. Graph 3-12 illustrates that 26% of the 145 Inpatient 
recommendations made by the panelists in the Independent Review actually received Inpatient 
treatment. The converse of this is that 74% of the clients recommended for Inpatient treatment 
actually received some other form of treatment. Twenty percent of the 76 cases recommended 
by the panelists for Intensive Outpatient placement actually received this treatment. While the 
majority cases, 69%, recommended for Regular Outpatient placements actually received Regular 
Outpatient treatment. Thus, most of the cases recommended for Inpatient and Intensive 
Outpatient treatment did not actually receive these treatments. 



GRAPH 3-12: PERCENT OF REALISTIC RECOMMENDATIONS THAT 
CORRESPOND WITH ACTUAL PLACEMENTS, IN THE 
INDEPENDENT REVIEW 
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Graph 3-13 illustrates the same information as that presented in Graph 3-12, but for ideal 
recommendations made in the Independent Review. A very similar relationship is evident. 
Again, 26% of the 173 clients recommended for Inpatient treatment, given ideal circumstances, 
were actually placed in Inpatient treatment; 17% of the recommendations for Intensive Outpatient 
actually received Intensive Outpatient treatment; and 77% of the recommendations for Regular 
Outpatient actually received Regular Outpatient treatment. 

The similarities between the graphs for realistic and ideal recommendations (Graphs 3-12 
and 3-13) are not surprising, given the high level of agreement between these two placement 
recommendations made by the panelists. Further analysis of the Team Review recommendations 
also reveals very similar findings. The small differences between realistic and ideal 
recommendations would also indicate that treatment availability probably accounts for only a 
small part of the difference between actual and recommended treatment. 



GRAPH 3-13: PERCENT OF IDEAL RECOMMENDATIONS THAT 
CORRESPOND WITH ACTUAL PLACEMENTS, IN THE 
INDEPENDENT REVIEW 
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Graphs 3- 14a and 3- 14b show, for the Independent review, how the clients recommended 
for a particular treatment were distributed for actual treatment. (Graph 3- 14a shows the 
distribution for realistic recommendations, and Graph 3- 14b shows the distribution for ideal 
recommendations.) Each pie chart illustrates the recommendations for a particular modality. In 
Graph 3- 14a the first pie chart, for Inpatient recommendations in the Independent Review, shows 
that 26% of the clients were actually placed in Inpatient treatment, while 22% actually received 
Intensive Outpatient treatment, 37% actually received Regular Outpatient treatment, and 15% 
received some other form of treatment (usually education). The second and third pie charts 
illustrate the same concept for cases recommended by the panelists for Intensive Outpatient 
treatment and Regular Outpadent treatment, respectively. 



GRAPH 3-14a: DISTRIBUTION OF ACTUAL PLACEMENTS FOR INDEPENDENT 
REALISTIC RECOMMENDATIONS 
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GRAPH 3-14b: DISTRIBUTION OF ACTUAL PLACEMENTS FOR INDEPENDENT 
IDEAL RECOMMENDATIONS 
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Correspondence Between Recommended Placements and Actual Treatment, For Cases With 
and Without PEI Results 



In the Independent Ideal Review, all panelists' recommendations were compared to the 
actual treatment received by the clients. The cases were divided by presence or absence of PEI 
results to determine the importance of the PEI in making placement decisions. As Graph 3-15 
reveals, in 24% of the 187 case reviews without PEI results, the placement recommendation and 
actual treatment was the same. When PEI results were available (in 156 case reviews) there was 
a higher level of correspondence between recommended and actual placements. This difference 
was statistically significant. Although the results reported in Graph 3-6 indicate the PEI does not 
influence the level of agreement, results reported in Graph 3-15 indicate that the PEI was, in fact, 
helpful in making placement decisions. 



GRAPH 3-15: 
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D: FACTORS CRITICAL FOR PLACEMENT DECISIONS 



For each case in the Independent Review, the panelists rank ordered the five most 
important factors influencing their placement decision. This was done separately for both their 
realistic and ideal placement decisions. In the Team Review, if all three panelists agreed on the 
three most important factors this was recorded. Otherwise, each panelist indicated one factor. 
For a list of all factors, see Appendix A. 

The observers in the debriefing, and the panelists in the group discussion following the 
Team Review, reported minimal attentior to discussing the factors in the Team Review. The 
majority of the teams simply recorded the top choice of each panelist. For this reason, the 
evaluation of the factors will focus largely on the Independent Review. 

Distribution of Factors 

For the Independent and Team Reviews, the selected factors were distributed by category. 
As Table 3-7 shows, there was very little difference in the distribution of selected factors for the 
Independent and Team Reviews. Drug History factors were cited most frequently as critical in 
making placement decisions. The categories Family History and High Risk Behavior were stated 
to be the next most important considerations for placement. The factors identified least often as 
critical related to Medical Concerns and Level of Personal Development. 

TABLE 3-7: DISTRIBUTION OF FACTORS 
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5% 
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Personal Development 
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2% 


Medical Concerns 
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3% 


2% 
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Other 


7% 
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8% 


8% 
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10% 


13% 


14% 


13% 


TOTAL 


100% 


100% 


100% 


100% 9 



Note: 15 panelists x 24 cases x 5 factors = 1800 total factors selected. 
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Comparison of Factors for Ideal and Realistic Placements 



Table 3-7 shows there was very little difference between the factors identified as most 
important for realistic and ideal placement. Drug History factors were identified most often, and 
Personal Development and Medical Concerns were identified If ast often. When the realistic and 
ideal factors were compared one-to-one for the Independent Review, there was a match 79% of 
the time. 

Selected Factors by Treatment Modality 

The factors identified as most important by the panelists were examined by treatment 
modality, for the Independent Review. The selected factors for all clients recommended for 
Inpatient treatment were grouped together, and then distributed by category. This was done for 
the other two treatment modalities as well. 

Graphs 3-16 and 3-17 present three pie charts, one for each treatment modality. Each pie 
chart shows the distribution of selected factors by category. (The pie charts, however, do not 
include the category "Other" and "No Response".) For this analysis, treatment placement for a 
case was defined as the placement selected by the panelists. Graph 3-16 illustrates critical factors 
for realistic placement in the Independent Review, and Graph 3-17 presents critical factors for 
ideal placement in the Independent Review. 



GRAPH 3-16: 
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GRAPH 3-17: FACTORS CITED, BY RECOMMENDED IDEAL PLACEMENTS, 
IN THE INDEPENDENT REVIEW 




Comparing the distribution of factors presented in Table 3-7 with the distribution of 
factors, by treatment modality, presented in the pie charts (Graphs 3-16 and 3-17), shows many 
similarities. Across the three modalities, Drug History factors are cited most frequently as being 
critical in making placement decisions. However, for both realistic and ideal placements, Drug 
History was cited more often for Inpatient placements than for Regular Outpatient placements. 
This difference was statistically significant for both rea'istic and ideal placements. (Note: The 
distribution of factors for which this was calculated differs from that presented in the pie charts. 
This is because "Other" responses and no responses were included in calculating the total.) 

Medical concerns, including such factors as pregnancy and drug-related medical problems, 
were cited infrequently for the three treatment modalities. It might be expected that clients 
would be placed more often in Inpatient treatment due to medical problems, but the data do not 
support this. 
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The Five Most Often Cited Factors for Each Modality 



As shown in Graphs 3-16 and 3-17, when factors were analyzed by categories, there was 
very little difference in the factors cited in each modality. To further examine possible 
differences between critical factors by modality, the five most important individual factors (not 
by category, but of the list of 41 factors) are extracted. 

Graph 3-18 shows, in order, the five most frequently cited factors when Inpatient was 
recommended. That is, in 82% of the cases recommended for Inpatient, "signs and symptoms" 
was a factor in making that recommendation. In fact, Graphs 3-19 and 3-20 show that "signs and 
symptoms" was also the most often cited factor for Intensive Outpatient and Regular Outpatient 
placement recommendations. However, it is also important to note that the most frequently cited 
factors do vary slightly for the different modalities. For instance, "school performance" was cited 
often for both outpatient modalities, but it was not for Inpatient. "Criminal involvement" was 
frequently cited for Inpatient and Regular Outpatient, but not as often in Intensive Outpatient. 



GRAPH 3-18: 



FACTORS CITED MOST OFTEN FOR INPATIENT 
RECOMMENDATIONS 



100% x 



§ 80% 
g 60% 

B 
E 

8 40% 



20% 



82% 



36% 





29% 


28% 














20% 




— i — 




= 









Signs & 
Symptoms 



Family 
Chemical 
Dependency 



Type of 
Support 
Syrtem 



Criminal 
Involvement 



Suicidal 
Tendency 



33 



42 



GRAPH 3-19: FACTORS CITED MOST OFTEN FOR 

INTENSIVE OUTPATIENT RECOMMENDATIONS 
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GRAPH 3-20: FACTORS CITED MOST OFTEN FOR REGULAR OUTPATIENT 
RECOMMENDATIONS 
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Through further analysis, more difference between the factors cited for each of the 
modalities were found. Although they were not the most often cited factors it is interesting to 
find that "attempts at abstinence" was considered more often in Inpatient (11%) and Intensive 
Outpatient (12%) than it was for Regular Outpatient (2%). Not surprisingly, a client's "history 
of running away" was a factor more often cited in making an Inpatient (12%) placement 
recommendation than in making either an Intensive Outpatient (2%) or a Regular Outpatient (4%) 
recommendation. Furthermore, "self-destructive behavior" proved to be more important for 
Inpatient (15%) than for Intensive Outpatient (9%) and Regular Outpatient (4%) treatment 
recommendations. 



34 




Factors Cited in Cases With and Without the Personal Experience Inventory (PEI) 

To further test the effects of the PEI on the placement decision process, the factors cited 
for cases with and without the PEI were analyzed. Graph 3-21 shows that there was generally 
very little difference between cases with and without the PEI in the factors cited (by category). 
A statistically significant difference observed was in the category of "Family History." Thi& 
category was cited in 20% of the cases with the PEI and in 15% of the cases without the PEL 
This may suggest that important information regarding a client's family history is often extracted 
with the use of the PEI. For the other categories the differences were not significant. 



GRAPH 3-21: FACTORS CITED BY CATEGORY FOR CASES WITH AND 
WITHOUT THE PEI 
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A: STUDY LIMITATIONS 



Several limitations to this study should be acknowledged when considering the results. 

One of the basic premises of the Peer Panel Review is that the panelists are representative 
of the substance abuse counselors in Washington State. As was discussed in Chapter II on Study 
Design, the panelists were very carefully selected, based on the six Regional Administrators' 
recommendations. However, of the fifteen panelists, eight were currently working as treatment 
program administrators. This over-selection of administrators probably had little effect on the 
study outcome, yet it should be recognized that they most likely have a greater breadth of 
experience than the average treatment counselor. 

The chief concern of the panelists regarding the study is that it is unrealistic to make 
treatment placement decisions based solely on written materials. The one-on-one interaction with 
the adolescent client provides a great deal of information that is impossible to document on 
paper. Thus, the panelists believed they were making their decisions based on a very limited 
amount of information. 

The assessment files reviewed differed significantly in their content. Some files contained 
three or four pages of scanty information, while others provided numerous pages of detailed 
material. In reviewing sr me of the more sketchy files, the observers reported a high level of 
frustration for the panelists, and sometimes less attention and time was spent on these cases. 
Often, more time was spent discussing the lack of information, than appropriate treatment. 

The observers also believed that some of the differences in jn< gment had to do with the 
moderators. The moderators had different styles of presenting informs don, managing discussions, 
and participating personally. 

Further evaluation of the observer reports identified other limitations. Initially, there was 
disagreement on the definitions of the treatment modalities. These differences were discussed 
within the teams, but not within the entire panel. Different teams may have been using different 
definitions. The observers also noted that some of the panelists appeared to view achieving 
consensus as a goal. This may have led to a higher level of agreement than might otherwise 
have been achieved. 
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B: IMPLICATIONS AND RECOMMENDATIONS 



The high level of correspondence between the proposed ideal and realistic placements 
implies that the panelists generally believe that the care they see as needed, by modality, is 
available in the areas where they work. No regional or metropolitan/non-metropolitan differences 
were found. However, there were differences between realistic and ideal placements regarding 
treatment durations. For Inpatient and Intensive Outpatient treatment, the mean recommended 
duration was significantly longer for ideal treatment than realistic treatment. This strongly 
suggests a need to increase available treatment length for these two modalities. Little difference 
in duration was found between realistic and ideal placements for Regular Outpatient clients. 

Most panelists expressed disappointment and surprise at the great variability in the 
different assessment forms. They felt some of the assessments were far too general and non- 
specific, and they were concerned the clients might be inappropriately placed due to a lack of 
detailed information being collected. When the observers' notes were examined, the most 
common remark was the lack of information available in the files. In general, the panelists felt 
stricter guidelines on what to include in an assessment form should be developed and 
implemented by DASA. Furthermore, some panelists felt a standardized assessment form (not 
the PEI) should be developed for use at all adolescent treatment facilities. 

Throughout the two-day Team Review, the panelists frequently mentioned the benefits 
gained from discussing and sharing their ideas regarding treatment. The panelists had the 
opportunity to discuss with their colleagues treatment alternatives, different philosophies, and 
treatment options available around the state. Many of the panelists left the meeting having 
learned about new and lesser-known resources available. In an anonymous evaluation completed 
after the meeting, 88% of the panelists stated more seminars and training geared toward 
standardizing assessments and intakes would be helpful. 

The adolescent substance abuse treatment field is still quite new. The lack of consistency 
regarding treatment placement that was present in this study strongly suggests the need for further 
training and education around assessment issues. At present, there is little standardization of the 
assessment process. Developing and imposing detailed guideline for doing assessments would 
help standardize and impiove the field of adolescent treatment. 
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C: FUTURE DIRECTIONS 



If this study were to be replicated, there are several methodological changes that might 
be helpful. 



Increase the number of panelists 

Common cases were used to determine the strength of significance found for level of 
agreement among all 15 panelists. However, using only 15 panelists made it difficult to obtain 
enough statistical power to examine significant agreement. Using a larger number of panelists 
would have made this analysis more reliable. Obtaining agreement among 30 panelists, rather 
than only 15, would provide a more reliable level of agreement. 



Revise the team review meeting 

The validity and value of the Team Review findings are questionable. The intent of 
having the panelists get together to discuss the cases was to determine if a higher level of 
agreement could be achieved than when they independently reviewed the cases. Though a higher 
level of agreement was achieved in the Team Review, the reasons for this are difficult to 
ascertain. As discussed in this report, it is possible that panelists agreed with each other because 
of the tendency to go along with the majority. The independent reviews more nearly represent 
the real world situation of client assessments than do team reviews. Team reviews would seem 
to be useful in future research mainly to gain insight into what experts might suggest as a group 
and to improve understanding of the elements involved in assessments. 

One valuable outcome of the Team Meeting was the general discussion with all panelists 
that followed the case review. They provided valuable insight into the placement ' 
process, both for us and for each other. This meeting also provided a time to learn about 
different adolescent treatment facilities available around the state. 



Modify PEI analysis 

Examination of the utility of the PEI for assisting in the placement process is very 
important. A more structured experiment would have been helpful. Future studies might have 
panelists evaluate cases excluding PEI summary data, and then later provide panelists with PEI 
results for the same cases. Comparing the same cases excluding, and then including, PEI data 
would provide greater certainty of the PEI's value. 
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Obtain actual counselors' placement recommendations 

This study compared panelists* recommendations with actual placements. This provides 
a measure of how closely actual placements match ideal recommendations. However, it would 
have been appropriate and informative to also compare actual recommendations with both actual 
placements and panelists' recommendations. 

Need more explicit term definitions 

One problem with the study is that the terms used were not always clear-y defined. 
Several observers reported the panelists spent time in the team meetings defining terns. Clear 
definitions should have been agreed upon prior to the independent assessments. This would 
include defining the three treatment modalities, and the difference between ideal and realistic 
conditions. Likewise, the 48 factors should have been clearly defined. Some were quite vague, 
such as "signs and symptoms" and "type of support system". Either could have been interpreted 
from a positive or negative standpoint. This led to ambiguity in the interpretation. 

Include education as a treatment option 

In the procedural meeting, the panelists reported three common treatment options: 
Inpatient, Intensive Outpatient, and Regular Outpatient. In reviewing the cases, however, the 
panelists occasionally recommended education as an alternative to treatment, particularly for those 
cases with minimal drug use. Had education been an option listed on the evaluation form, the 
panelists may have selected it even more frequently. 

Provide training for the moderators 

The observers reported a great deal of variation in how the moderators led the discussions. 
Some were actively involved in directing the conversations, while others facilitated with a more 
objective approach. It is likely that this variation led to differences in focus, and possibly it 
influenced the level of agreement achieved. More consistency between the moderators may have 
been achieved if they had received training on ho;v to conduct the discussions. Rather than using 
specialists from the adolescent substance abuse field, it may have been more helpful to have 
researchers serve as moderators. 

Use observers familiar with the field 

The observers reported difficulty following the discussions. This difficulty may have been 
reduced if the observers were more familiar with the field of adolescent treatment. A conscious 
decision was made to use observers from outside the field as a way of increasing their 
objectivity. However their lack of familiarity with the terms and jargons used, and issues 
discussed, impeded their ability to thoroughly report on the discussions. Using researchers as 
observers might overcome these obstacles. 
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APPENDIX A: FACTORS THE PANELISTS COULD SELECT ON THE 
EVALUATION FORM 



DRUG HISTORY: 


SOCIAL BACKGROUND: 


Prior treatment experience 


School performance 


Length of use 


Friends using 


Type of drugs used 


Client in gang 


Age of first use 


Physical violence common 


Last use 


Social functioning 


Frequency of use 


Peer relations 


Method of use 




Signs and Symptoms 




Attempts at abstinence 


DEVELOPMENT: 


Tolerance level 




Pattern of use over time 


Level of maturity 




Client s age 




Educational level 


FAMILY HISTORY: 


Developmental disability 




Learning disability 


Type of support system 




Physical abuse 




Emotional abuse 


HIGH RISK BEHAVIOR: 


Sexual abuse 




Family chemical dependency 


History of running away 


Person in household 


Criminal involvement 




Suicidal tendency 




Assaultive behavior 


MEDICAL CONCERNS: 


Self-destructive behavior 


Drug-related medical problems 




Eating disorders 


OTHER: 


Medical problems 




Pregnant client 


Mental health problems 


Medication history 


Motivation of client toward treatment 




Client's own statement re: problem 




APPENDIX B: TREATMENT FACILITIES THAT PROVIDED CASES 

Client cases were received from the following agencies for use in the Peer Panel Review: 



ALPHA HOUSENFORCE 
4214 PORTLAND AVE 
TACOMAWA 98404 

AUBURN YOUTH SERVICES 
816 F STREET SE 
AUBURN WA 98002 

CASCADE RECOVERY SERVICES 
5630 S 296TH COURT 
AUBURN WA 98001 

CENTER FOR HUMAN SERVICES 
17011 MERIDIAN AVE N 
SEATTLE WA 98133 

CENTRAL YOUTH & FAMILY SERVICES 
1730 BRADNER PLACE S 
SEATTLE WA 98144 

COMMUNITY ALCOHOL & DRUG CENTER 
1260 COMMERCE SUITE 2 
LONGVIEWWA 98532 

CONSEJO 

3808 S ANGELINE ST 
SEATTLE WA 98118 

DAYBREAK OF SPOKANE 
POST OFFICE BOX 8616 
SPOKANE WA 99203 

405 PROGRAM 

1206 S 11TH ST STE 8 

TACOMA WA 98405 

NEW BEGINNINGS OF NW 
600 NORTH 130TH 
SEATTLE WA 98133 

CHILD & FAMILY SERVICES 
KITSAP MENTAL HEALTH SERVICES 
500 UNION 

BREMERTON WA 98312 

PROJECT YOUTH RECOVERY 
BENTON-FRANKLIN JUVENILE COURT 
5606 W CANAL PLACE SUITE 106 
KENNEWICK WA 99336 



OLYMPIC COUNSELING CENTER 
1215 REGENTS BLVD SUITES A & B 
TACOMAWA 98466 

PACIFIC TREATMENT ALTERNATIVES 
1114 PACIFIC AVENUE 
EVERETT WA 98201 

PARKE CREEK 
ROUTE 3 BOX 1 120 
ELENSBURG WA 98926 

PIERCE COUNTY ALLIANCE 
710 SOUTH FAWCETT 
TACOMA WA 98402 

RECOVERY NORTHWEST 
391 NW STATE AVENUE 
CHEHALIS WA 98532 

RYTHER CHILD CENTER 
2400 NORTHEAST 95TH 
SEATTLE WA 98115 

ST PETER'S CHEMICAL DEPENDENCY 
4800 COLLEGE STREET SE 
LACEYWA 98503 

THURSTON-MASON COMM MENTAL HEALTH 
112 EAST STATE STREET 
OLYMPIA WA 98501 

OLYMPIC CENTER 
1603 E ILLINOIS 
BELLINGHAM WA 98226 

TOUTLE RIVER BOYS RANCH 
2232 S SILVERDALE RD 
CASTLE ROCK W A 98611 

RUTH DYKEMAN CHILDREN'S CENTER 
YOUTH & FAMILY SERVICES BRANCH 
POST OFFICE BOX 66086 
SEATTLE WA 98115 
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APPENDIX C: LEVEL OF PROFESSIONAL AGREEMENT AMONG PANELISTS 
(Unweighted Values) 

When analyzing consensus it is important to take into account that the probability of 
reaching complete agreement is much smaller than reaching majority or no agreement in a team 
of three. There are only 4 outcomes that would result in a complete agreement: all three 
recommend either Inpatient, Intensive Outpatient, Regular Outpatient, or Other (usually 
education). However, a team of three can have a majority agreement with 16 different outcomes 
(for example, 2 IP with 1 IOP, or 2 IP with 1 ROP). Furthermore, there are 15 possible ways 
a team of three can result in "no agreement." Because the probability of reaching each type of 
agreement is not the same (IP:IOP;ROP = 4:16:15), weighted values are more appropriate. The 
raw percentages are presented here to show the effects of the weighting procedure. 

Compared to the weighted values in Graph 3-1, Graph D-l shows that the rate of majority 
agreement was higher in the Independent Review for both realistic (51%) and ideal (45%) 
placements, while complete agreement was lower. In the Team Review, complete agreement was 
the result most often found for both realistic (62%) and ideal (67%) placements. 



GRAPH C-la: LEVEL OF PROFESSIONAL AGREEMENT (UNWEIGHTED) IN 
PLACEMENTS 
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GRAPH (Mb: 
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Comparison of Agreement in fhe Independent and Team Reviews 

Complete agreement increased more dramatically from Independent to Team Review when 
the unweighted numbers were analyzed. In the Independent Review, there was 23% (realistic) 
and 28% (ideal) complete agreement which increased substantially to 62% (realistic) and 67% 
(ideal) in the Team Review (Graph D-2). While the trend of increasing complete agreement is 
evident in weighted and unweighted numbers, the unweighted numbers give a greater value to 
this increase. 



GRAPH C-2: COMPARISON OF RATES OF COMPLETE AGREEMENT 

AMONG PANELISTS IN PLACEMENT RECOMMENDATIONS 
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Agreement for Cases With and Without the Personal Experience Inventory (PEI) 

When consensus is analyzed with regard to the PEI, the results must be weighted 
according to the probability of attaining a particular level of agreement. Graph C-3 shows the 
unweighted findings before weighting them according to the probabilities. 



GRAPH C-3: 



LEVEL OF AGREEMENT FOR IDEAL PLACEMENT AMONG 
TEAM MEMBERS FOR CASES WITH AND WITHOUT THE PEI 
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APPENDIX E: 



SIGNIFICANCE TESTS USED IN REPORT 



Figure 


Analysis 


Test Used 


Values Obtained | 

" — " 


Graph 3-3 


From Independent to Team Review, 
Increase In Consensus (Weighted), 
General 


Modified t test for 
the difference in 
proportions 


Realistic, Z=5.75 (sig) 
Ideal, Z=5.00(sig) 


Graph 3-6 


In Independent & Team Reviews, 
Level Of Consensus (Weighted), 
PEI v No PEI 


Modified t test for 
the difference in 
proportions, and 
Chi Square test 


Independent, Z=1.38 (ns) 


Graph 3-8 


In Independent Review, 
Correspondence Between Realistic 
& Ideal, East v West 


Modified t test for 
the difference in 
proportions 


Z=1.66 (ns) 


Graph 3-10 


For each of the Common Cases, the 
Level of Agreement was Tested for 
Significance 


Modified t test for 
the difference in 
proportions 


Values determined for each case, 
for Independent and Team 
Reviews, for both Ideal and 
Realistic placements 


Graph 3-12 


Percentage of Ideal 
Recommendations that correspond 
to Actual Placements in the 
Independent Review was higher for 
ROP than IP and IOP 


Modified t test for 
the difference in 
proportions 


IP v IOP, Z= 1.46 (ns) 
IP v ROP, Z=6.65 (sig) 
IOP v ROP, Z=6.50 (sig) 


Graph 3-13 


Percentage of Realistic 
Recommendations that correspond 
to Actual Placements in the 
Independent Review was higher for 
ROP than IP and IOP 


Modified t test for 
the difference in 
proportions 


IP v IOP, Z=0.99 (ns) 
IP v ROP, Z=6.15 (sig) 
IOP v ROP, Z=6.06 (sig) 


Graph 3-15 


Percentage of cases for which 
Independent ideal recommendation 
and actual placements were the 
same, (higher for PEI cases) 


Modified t test for 
the difference in 
proportions 


Z=2.24(sig) 


Table 3-2 


From Realistic to Ideal Placement, 
increase in Mean uurauon, oy 
Modality 


Standard t test 


IP, t=8.50 (sig) 

iur, t— i.y j (sig-i tanec test) 

ROP, t=0.70 (ns) 


Graph 3-16 
and 3-17 


Factors, Drug History Cited More 
Frequently for IP than IOP and ROP 


Modified t test for 
the difference in 
proportions 


Realistic: 
IP v ROP, Z=2.60 (sig) 
IOP v ROP, Z=1.43 (ns) 
IP v IOP, Z=0.96(ns) 

Ideal: 

IP v ROP, Z=2.64(sig) 
IOPvROP,Z=1.78(ns) 
IP v IOP, Z=0.63 (ns) 


Graph 3-21 


The presence of the PEI has an 
effect on the frequency "Family 
History", "Social Background", and 
"Drug History" were cited 


Chi Squared Test 


Family History, Chi Sq.=7.62 (sig) 
Social Backgd, Chi Sq.=2.66 (ns) 
Drug History, Chi Sq.=0.44 (ns) 
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